Análisis de tweets de @brasilsemaborto

Información

Los datos de este usuario cubren desde la creación de la cuenta 2009-07-31 hasta 2023-01-01

Code

df.info()

<class 'pandas.core.frame.DataFrame'>
Index: 1411 entries, 199416 to 200826
Data columns (total 63 columns):
 #   Column                   Non-Null Count  Dtype                            
---  ------                   --------------  -----                            
 0   query                    1411 non-null   object                           
 1   id                       1411 non-null   float64                          
 2   timestamp_utc            1411 non-null   int64                            
 3   local_time               1411 non-null   object                           
 4   user_screen_name         1411 non-null   object                           
 5   text                     1411 non-null   object                           
 6   possibly_sensitive       623 non-null    object                           
 7   retweet_count            1411 non-null   float64                          
 8   like_count               1411 non-null   float64                          
 9   reply_count              1411 non-null   float64                          
 10  impression_count         10 non-null     object                           
 11  lang                     1411 non-null   object                           
 12  to_username              281 non-null    object                           
 13  to_userid                281 non-null    float64                          
 14  to_tweetid               254 non-null    float64                          
 15  source_name              1411 non-null   object                           
 16  source_url               1411 non-null   object                           
 17  user_location            1411 non-null   object                           
 18  lat                      9 non-null      object                           
 19  lng                      9 non-null      object                           
 20  user_id                  1411 non-null   object                           
 21  user_name                1411 non-null   object                           
 22  user_verified            1411 non-null   float64                          
 23  user_description         1411 non-null   object                           
 24  user_url                 1411 non-null   object                           
 25  user_image               1411 non-null   object                           
 26  user_tweets              1411 non-null   object                           
 27  user_followers           1411 non-null   float64                          
 28  user_friends             1411 non-null   object                           
 29  user_likes               1411 non-null   float64                          
 30  user_lists               1411 non-null   float64                          
 31  user_created_at          1411 non-null   object                           
 32  user_timestamp_utc       1411 non-null   float64                          
 33  collected_via            1411 non-null   object                           
 34  match_query              1411 non-null   float64                          
 35  retweeted_id             0 non-null      float64                          
 36  retweeted_user           0 non-null      float64                          
 37  retweeted_user_id        0 non-null      float64                          
 38  retweeted_timestamp_utc  0 non-null      object                           
 39  quoted_id                6 non-null      object                           
 40  quoted_user              6 non-null      object                           
 41  quoted_user_id           6 non-null      float64                          
 42  quoted_timestamp_utc     6 non-null      float64                          
 43  collection_time          1411 non-null   object                           
 44  url                      1411 non-null   object                           
 45  place_country_code       15 non-null     object                           
 46  place_name               15 non-null     object                           
 47  place_type               15 non-null     object                           
 48  place_coordinates        15 non-null     object                           
 49  links                    433 non-null    object                           
 50  domains                  433 non-null    object                           
 51  media_urls               326 non-null    object                           
 52  media_files              326 non-null    object                           
 53  media_types              326 non-null    object                           
 54  media_alt_texts          21 non-null     object                           
 55  mentioned_names          207 non-null    object                           
 56  mentioned_ids            197 non-null    object                           
 57  hashtags                 638 non-null    object                           
 58  intervention_type        0 non-null      float64                          
 59  intervention_text        0 non-null      float64                          
 60  intervention_url         0 non-null      float64                          
 61  country                  1411 non-null   object                           
 62  date                     1411 non-null   datetime64[ns, America/Sao_Paulo]
dtypes: datetime64[ns, America/Sao_Paulo](1), float64(20), int64(1), object(41)
memory usage: 705.5+ KB

Datos

Code

# count items on column
domains_list = df['domains'].value_counts()

# return first n rows in descending order
top_domains = domains_list.nlargest(20)

top_domains

domains
bit.ly                                                                    94
brasilsemaborto.org                                                       87
youtube.com                                                               30
youtu.be                                                                  29
wp.me                                                                     28
instagram.com                                                             24
facebook.com                                                              19
twitpic.com                                                               19
brasilsemaborto.com.br                                                    11
gazetadopovo.com.br                                                        8
camara.leg.br                                                              8
www12.senado.gov.br                                                        8
twitpic.com|twitpic.com                                                    4
fb.me                                                                      3
itaucinemas.com.br                                                         3
brasilsemaborto.wordpress.com                                              3
veja.abril.com.br                                                          3
brasilsemaborto.org|twitter.com|facebook.com|instagram.com|youtube.com     3
noticias.cancaonova.com                                                    3
www12.senado.leg.br                                                        3
Name: count, dtype: int64

Hashtags

Lista del top 20 de hashtags más usados y su frecuencia

Code

# convert dataframe column to list
hashtags = df['hashtags'].to_list()

# remove nan items from list
hashtags = [x for x in hashtags if not pd.isna(x)]

# split items into a list based on a delimiter
hashtags = [x.split('|') for x in hashtags]

# flatten list of lists
hashtags = [item for sublist in hashtags for item in sublist]

# count items on list
hashtags_count = pd.Series(hashtags).value_counts()

# return first n rows in descending order
top_hashtags = hashtags_count.nlargest(20)

top_hashtags

brasilsemaborto          271
marchavirtualpelavida     87
codigopenal               73
stfabortonao              71
pelas2vidas               61
mulhersimabortonao        35
marchapelavida            33
abortoépreconceito        33
asduasvidasimportam       33
10anos                    30
estatutodonascituro       28
afavordavida              24
brasilpelasduasvidas      16
verdadepelavida           16
avidadependedoseuvoto     15
simàvida                  12
diadonascituro            10
anencefalo                 9
avidaporumfio              9
stfdiganaoaoaborto         7
Name: count, dtype: int64

Usuarios

Top 20 de usuarios más mencionados en los tweets

Code

# filter column from dataframe
users = df['mentioned_names'].to_list()

# remove nan items from list
users = [x for x in users if not pd.isna(x)]

# split items into a list based on a delimiter
users = [x.split('|') for x in users]

# flatten list of lists
users = [item for sublist in users for item in sublist]

# count items on list
users_count = pd.Series(users).value_counts()

# return first n rows in descending order
top_users = users_count.nlargest(20)

top_users

lenisegarcia       41
brasilsemaborto    29
addthis             8
rebeccakiesslin     6
stf_oficial         6
anabeatrizries      5
mpf_pgr             4
cnnoticias          4
luh_lena            4
anadep_brasil       4
gazetadopovo        4
jorgeferraz         4
veja                3
alosenado           3
wagnermoura         3
angela_gandra       3
jornaldacbn         3
addtoany            2
agenciacamara       2
eunicio             2
Name: count, dtype: int64

Likes en el tiempo

Code

# plot the data using plotly
fig = px.line(df, 
              x='date', 
              y='like_count', 
              title='Likes over Time',
              template='plotly_white', 
              hover_data=['text'])

# show the plot
fig.show()

Tokens

Lista del top 20 de los tokens más comunes y su frecuencia

Code

# load the spacy model for Portuguese
nlp = spacy.load("pt_core_news_sm")

# load stop words for Spanish
STOP_WORDS = nlp.Defaults.stop_words

# Function to filter stop words
def filter_stopwords(text):
    # lower text
    doc = nlp(text.lower())
    # filter tokens
    tokens = [token.text for token in doc if not token.is_stop and token.text not in STOP_WORDS and token.is_alpha]
    return ' '.join(tokens)

# apply function to dataframe column
df['text_pre'] = df['text'].apply(filter_stopwords)

# count items on column
token_counts = df["text_pre"].str.split(expand=True).stack().value_counts()[:20]

token_counts

vida                     494
aborto                   306
brasilsemaborto          274
marcha                   165
brasil                   131
nacional                 110
dia                      109
marchavirtualpelavida     88
movimento                 85
nascituro                 77
participe                 74
codigopenal               73
mãe                       71
defesa                    71
stfabortonao              70
hoje                      66
estatuto                  65
código                    64
saiba                     63
acompanhe                 60
Name: count, dtype: int64

Hora

Lista de las 10 horas con más cantidad de tweets publicados

Code

# extract hour from datetime column
df['hour'] = df['date'].dt.strftime('%H')

# count items on column
hours_count = df['hour'].value_counts()

# return first n rows in descending order
top_hours = hours_count.nlargest(10)

top_hours

hour
11    193
10    161
16    130
15    128
19    111
12    106
09     94
17     91
14     72
18     57
Name: count, dtype: int64

Pataformas

Plataformas desde las que se publicaron contenidos y su frecuencia

Code

df['source_name'].value_counts()

source_name
Twitter for Android            479
Twitter Web Client             453
Plume for Android              161
Twitter Web App                 89
Twitter for iPhone              60
TweetDeck                       42
Jetpack.com                     41
Twitter for Websites            26
TweetCaster for Android         18
Posterous                       16
Gravity                         15
Twitter for Android Tablets      7
Gravity!                         2
Twibbon                          1
Twitpic                          1
Name: count, dtype: int64

Tópicos

Técnica de modelado de tópicos con transformers y TF-IDF

Code

# visualize topics
topic_model.visualize_topics()

Reducción de tópicos

Mapa con 10 tópicos del contenido de los tweets

Code

# visualize topics
topic_model.visualize_topics()

Términos por tópico

Code

topic_model.visualize_barchart(top_n_topics=11)

Análisis de tópicos

Selección de tópicos que tocan temas de género

Code

# selection of topics
topics = [0, 2]

keywords_list = []
for topic_ in topics:
    topic = topic_model.get_topic(topic_)
    keywords = [x[0] for x in topic]
    keywords_list.append(keywords)

# flatten list of lists
word_list = [item for sublist in keywords_list for item in sublist]

# use apply method with lambda function to filter rows
filtered_df = df[df['text_pre'].apply(lambda x: any(word in x for word in word_list))]

percentage = round(100 * len(filtered_df) / len(df), 2)
print(f"Del total de {len(df)} tweets de @brasilsemaborto, alrededor de {len(filtered_df)} hablan sobre temas de género, es decir, cerca del {percentage}%")

Del total de 1411 tweets de @brasilsemaborto, alrededor de 1020 hablan sobre temas de género, es decir, cerca del 72.29%

Code

# drop rows with 0 values in two columns
filtered_df = filtered_df[(filtered_df.like_count != 0) & (filtered_df.retweet_count != 0)]

# add a new column with the sum of two columns
filtered_df['impressions'] = (filtered_df['like_count'] + filtered_df['retweet_count'])/2

# extract year from datetime column
filtered_df['year'] = filtered_df['date'].dt.year

# remove urls, mentions, hashtags and numbers
p.set_options(p.OPT.URL)
filtered_df['tweet_text'] = filtered_df['text'].apply(lambda x: p.clean(x))

# Create scatter plot
fig = px.scatter(filtered_df, x='like_count', 
                 y='retweet_count',
                 size='impressions', 
                 color='year',
                 hover_name='tweet_text')

# Update title and axis labels
fig.update_layout(
    title='Tweets talking about gender with most Likes and Retweets',
    xaxis_title='Number of Likes',
    yaxis_title='Number of Retweets'
)

fig.show()

Tópicos en el tiempo

Code

# convert column to list
tweets = df['text_pre'].to_list()
timestamps = df['local_time'].to_list()

topics_over_time = topic_model.topics_over_time(docs=tweets, 
                                                timestamps=timestamps, 
                                                global_tuning=True, 
                                                evolution_tuning=True, 
                                                nr_bins=20)

topic_model.visualize_topics_over_time(topics_over_time, top_n_topics=20)